Cross Validation runs with images from 73 videos

• We have 43536 images with labels from 73 videos (including Clement Ave video).
• I created 2 sets of data:
  • 95% training data and 5% test data split
  • 90% training data and 10% test data split
  • from skmultilearn.model_selection import iterative_train_test_split (a special library for multilabel stratified splitting) was used to do the split.
• The training data were further split into training and validation datasets using from iterstrat.ml_stratifiers import MultilabelStratifiedKFold.
• Outputs:
  • 10 sets of training and validation data generated from 95/5 split training dataset
  • 10 sets of training and validation data generated from 90/10 split training dataset

Summary Tables

The tables below show the average validation and test F1 normal and average F2 without ND scores of the 10 fold runs. 90/10 split dataset are used for training/validation and testing.

Using weight from previous model

Model	Val F1 normal	Test F1 normal	Val F2 avg no ND	Test F2 avg no ND
Baseline	0.989	0.987	0.974	0.971
Image augmentation	0.985	0.987	0.979	0.972
Weighted Loss	0.943	0.945	0.959	0.940

Train from scratch

Model	Val F1 normal	Test F1 normal	Val F2 avg no ND	Test F2 avg no ND
Baseline	0.975	0.970	0.949	0.904
Image augmentation	0.967	0.965	0.942	0.910
Weighted Loss	0.833	0.839	0.883	0.780

Baseline model

The parameters of the baseline model:

• Backbone: ResNet50
• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros')]
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5
• Weight from the previous model is used as an initial weight

90(train-val)/10(test) split

• Train/Validation Dataset: 90% of 43536 images (73 videos)
• Test Dataset: 10% of 43536 images (73 videos)
• Image labelling Method: B (Access database)

no	k fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_0	0	0.986	0.980	0.977	0.971
cv_1	1	0.988	0.990	0.969	0.974
cv_2	2	0.991	0.985	0.971	0.966
cv_3	3	0.989	0.989	0.986	0.971
cv_4	4	0.986	0.985	0.963	0.971
cv_5	5	0.990	0.988	0.974	0.968
cv_6	6	0.987	0.987	0.976	0.972
cv_7	7	0.990	0.991	0.982	0.985
cv_8	8	0.990	0.987	0.975	0.961
cv_9	9	0.988	0.987	0.970	0.971
Average		0.989	0.987	0.974	0.971

95(train-val)/5(test) split

• Train/Validation Dataset: 95% of 43536 images (73 videos)
• Test Dataset: 5% of 43536 images (73 videos)
• Image labelling Method: B (Access database)

no	k fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_10	0	0.990	0.995	0.986	0.960
cv_11	1	0.986	0.993	0.956	0.937
cv_12	2	0.991	0.991	0.990	0.969
cv_13	3	0.990	0.988	0.978	0.957
cv_14	4	0.990	0.992	0.969	0.953
cv_15	5	0.989	0.990	0.985	0.955
cv_16	6	0.990	0.990	0.979	0.964
cv_17	7	0.990	0.991	0.978	0.959
cv_18	8	0.989	0.990	0.955	0.950
cv_19	9	0.988	0.990	0.971	0.950
Average		0.989	0.991	0.975	0.955

Image Augmentation

The parameters of the model:

• Backbone: ResNet50
• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros')]; [Flip(),Brightness(), Warp(), Rotate()]
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5
• Weight from the previous model is used as an initial weight

90(train-val)/10(test) split

• Train/Validation Dataset: 90% of 43536 images (73 videos)
• Test Dataset: 10% of 43536 images (73 videos)
• Image labelling Method: B (Access database)

no	K fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_img_0	0	0.985	0.981	0.978	0.971
cv_img_1	1	0.977	0.987	0.967	0.966
cv_img_2	2	0.981	0.987	0.974	0.963
cv_img_3	3	0.993	0.988	0.989	0.980
cv_img_4	4	0.987	0.988	0.982	0.980
cv_img_5	5	0.983	0.989	0.977	0.968
cv_img_6	6	0.983	0.989	0.978	0.963
cv_img_7	7	0.991	0.989	0.987	0.979
cv_img_8	8	0.984	0.988	0.977	0.970
cv_img_9	9	0.986	0.988	0.981	0.980
Average		0.985	0.987	0.979	0.972

Weighted loss function

The parameters of the model:

• Backbone: ResNet50
• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros')]; [Flip(),Brightness(), Warp(), Rotate()]
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5
• Weight from the previous model is used as an initial weight
• Weighted loss function. The rare labels have bigger weight that the common labels

90(train-val)/10(test) split

no	K fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_wl_0	0	0.9452	0.942967323	0.957	0.932
cv_wl_1	1	0.9477	0.9444	0.955	0.946
cv_wl_2	2	0.9251	0.9303	0.943	0.928
cv_wl_3	3	0.9533	0.9535	0.979	0.954
cv_wl_4	4	0.9373	0.9382	0.953	0.928
cv_wl_5	5	0.9463	0.9485	0.964	0.933
cv_wl_6	6	0.9493	0.9554	0.960	0.957
cv_wl_7	7	0.9401	0.9419	0.972	0.952
cv_wl_8	8	0.9461	0.9491	0.956	0.940
cv_wl_9	9	0.9428	0.9461	0.951	0.930
Average		0.943	0.945	0.959	0.940

Train from scratch (pretrain=False)

The parameters of the model:

• Backbone: ResNet50
• Set pretrain=False (weight of the previous model will not be used)
• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros')]
• Batch Size: 32
• Epoch: 10
• Metric threshold = 0.5
• dataset = 90(train-val)/10(test) split
• Image labelling Method: B (Access database)

Baseline

• no additional parameters

no	K fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_nopre_0	0	0.975	0.972	0.956	0.922
cv_nopre_1	1	0.973	0.970	0.930	0.904
cv_nopre_2	2	0.975	0.966	0.945	0.902
cv_nopre_3	3	0.974	0.971	0.949	0.910
cv_nopre_4	4	0.976	0.968	0.948	0.906
cv_nopre_5	5	0.974	0.965	0.956	0.905
cv_nopre_6	6	0.973	0.973	0.948	0.897
cv_nopre_7	7	0.977	0.970	0.965	0.897
cv_nopre_8	8	0.975	0.967	0.944	0.892
cv_nopre_9	9	0.975	0.973	0.947	0.908
Average		0.975	0.970	0.949	0.904

Image augmentation

• Image argumentation: '[Resize((470, 700),method='squish', pad_mode='zeros')]; [Flip(),Brightness(), Warp(), Rotate()]

no	K fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_nopre_img_0	0	0.961	0.959	0.958	0.909
cv_nopre_img_1	1	0.970	0.966	0.940	0.918
cv_nopre_img_2	2	0.968	0.961	0.940	0.909
cv_nopre_img_3	3	0.965	0.962	0.917	0.893
cv_nopre_img_4	4	0.965	0.961	0.944	0.916
cv_nopre_img_5	5	0.969	0.967	0.943	0.925
cv_nopre_img_6	6	0.970	0.967	0.943	0.899
cv_nopre_img_7	7	0.971	0.970	0.964	0.916
cv_nopre_img_8	8	0.966	0.968	0.947	0.899
cv_nopre_img_9	9	0.968	0.969	0.927	0.913
Average		0.967	0.965	0.942	0.910

Weighted loss function

no	K fold	Validation Score (F1 normal	Test Score (F1 normal	Validation Score (F2 avg no ND)	Test Score (F2 avg no ND)
cv_nopre_wl_0	0	0.837	0.849	0.908	0.809
cv_nopre_wl_1	1	0.792	0.800	0.864	0.777
cv_nopre_wl_2	2	0.833	0.858	0.900	0.786
cv_nopre_wl_3	3	0.841	0.858	0.877	0.796
cv_nopre_wl_4	4	0.853	0.868	0.881	0.678
cv_nopre_wl_5	5	0.826	0.855	0.877	0.781
cv_nopre_wl_6	6	0.849	0.823	0.880	0.780
cv_nopre_wl_7	7	0.840	0.853	0.901	0.788
cv_nopre_wl_8	8	0.833	0.828	0.895	0.829
cv_nopre_wl_9	9	0.826	0.803	0.849	0.774
Average		0.833	0.839	0.883	0.780